Database mining system and method for coverage analysis of functional verification of integrated circuit designs

ABSTRACT

Database mining, analysis and optimization techniques in conjunction with the model-based functional coverage analysis are used to turn raw verification and coverage data into design intelligence (DI) and verification intelligence (VI). The required data and attributes are automatically extracted from verification, simulation and coverage analysis databases. Design finite state machine extraction, design functional event extraction, and automatic coverage model generation and optimization techniques are applied to the design HDL description. Coverage model tuning and optimization directives, as well as test spec tuning and optimization directives are generated based on the analysis and mining of various verification, simulation, and coverage databases. An integrated web-based interface portlet is used for access, analysis and management of the resulting databases, generated reports and verification directives. Dissemination rules are used to automatically generate and distribute analysis reports and verification directives to engineers at wired or wireless interface devices via Internet or Intranet.

FIELD OF THE INVENTION

This invention relates to analysis and verification of functionalfeatures of complex integrated circuit designs, such as processor cores,embedded microprocessors, and high performance microprocessors.

BACKGROUND OF THE INVENTION

Microprocessor verification is an important part and the bottleneck ofthe microprocessor design process. There is no practical method to testand verify all functional states of a complex microprocessor. A varietyof techniques are used to generate test programs (i.e. tests) and to usethese to verify the design's intended functionality. These includedeterministic tests manually generated by engineers, tests generated byautomatic test generators such as IBM's Genesys and GenesysPro randomtest generators, standard test benches or a combination of alltechniques. In all cases, test programs are simulated against the designmodel to identify faulty functions.

One major issue that must be addressed is whether all functions of theprocessor have been tested and/or verified. If not, which ones have beenmissed. This information should be available in a timely manner soresources can be directed to verify functions that have been missed.

It is also very important to know promptly which functions have alreadybeen verified so that duplications are reduced and resources aretargeted towards untested functions. Verification resources includeverification engineers, test generators, simulation computers' capacity(i.e. simulation cycles) and debugging resources, just to name a few.

As the complexity of integrated circuit designs increase, so do thenumber and complexity of test programs to be generated and simulated.This in turn requires more resources to simulate and debug the functionsthat fail. The following questions need to be answered in order todecide whether the design has been verified or not:

-   -   1) Has everything detailed in the test plan been verified. If        not, what are the remaining functions to be verified (functional        coverage holes);    -   2) Have all new scenarios/states not planned for in the test        plan been verified; and    -   3) Where should the attention/resources be focused to ensure the        best coverage of verification.

To answer these questions, functional coverage tools and methodologieshave been developed that monitor the simulation processes and analyzethe test programs, thereby generating a vast amount of data on what hasbeen tested so far. In addition, some coverage analysistools/methodologies allow the verification procedures to define modelsof classes of scenarios to be verified and/or enumerate all designstates to be verified. These in turn results in “coverage holes”reports, which can tell what scenarios/states, have not yet beensimulated/verified. Verification procedures can then focus on theremaining verification holes.

The simulation process generates a large amount of data, which needs tobe reviewed, and processed each day on a continuous basis (24/7).Automatic random test generators running on a large network of computerscan generate hundreds of thousands of tests each day resulting inseveral gigabytes of test data that must be filtered and analyzed eachday. In addition, coverage models and analysis filters applied to theabove tests generate gigabytes of functional coverage data each day. Inmany cases, the simulation data that is generated and its correspondingcoverage data can not be fully analyzed as fast as they are generated,resulting in:

-   -   1) A failure to extract all of the important coverage data from        simulation database, thereby having an incomplete view of what        has been simulated;    -   2) A failure to properly interpret the accumulated data to        provide correct feed back to verification process based on        accurate and complete coverage data;    -   3) Shallow analysis of simulated tests because coverage        models/filters are not extensive enough nor do they deal with        the tiered and temporal functional features and design        constraints;    -   4) Many interesting functional scenarios and patterns go        undetected because they are not reflected in the coverage models        and hence not detected;    -   5) No data is generated or considered on correlations between        models, since models are developed independently and coverage        analysis tools and methods work with the output of models and no        cross-model analysis capability is usually provided;    -   6) No new discovery is made based on the actual simulation or        coverage data because the generated tests or coverage data are        not analyzed as a whole from a temporal point of view or overall        design progress point of view but rather in isolation one test        at a time and one model at a time; and    -   7) Not all design changes are reflected in coverage models,        since models are defined based on the test plans and design        specs rather than the actual design itself.

Consequently, the verification bottleneck is now spread to test andcoverage analysis as well. Not only is the verification an incompletetask, but also the coverage analysis is incomplete and requiresadditional resources of its own.

Focused and targeted verification coverage analysis requires dynamicadjustment of the analysis process based on a deep knowledge of testfeatures and coverage analysis data, taking into consideration thetrends and relevant patterns of the actual verification progress, aswell as any known and expected patterns missing in the collected data.Unfocused coverage analysis generates reams of data and many shallowcoverage reports, which usually do not reflect the actual attributes ofthe verification process. Expertise for development of coverage analysismodels and effective utilization of coverage tools are scarce. Designersand verification engineers who are the best candidates to writeeffective coverage models and analyze coverage data because of theirknowledge of the design's internal workings and details are usually toobusy with their own traditional tasks. Therefore, a new and dynamiccoverage analysis process is required.

A way is needed to convert gigabytes of data collected each day touseable knowledge. In addition, hidden attributes, trends, patterns,associations and relationships in the collected verification andcoverage analysis data that may shed light on the actual behavior andprogress of the verification process is usually not considered. Coverageanalysis tools generate a variety of reports, presenting predefinedcoverage information, which usually show accumulated statistics ratherthan shifts in the trends or new behavior. Intelligent and methodicalanalysis based on deep knowledge of the verification process is requiredto turn continuous flow of test and coverage analysis data to readilyusable and relevant information. What is required is design intelligence(DI) and verification intelligence (VI) rather than more data and morereports.

BRIEF DESCRIPTION OF THE INVENTION

Among the objectives of the present invention is the ability to analyzetests generated by any and all means and identify whether all designattributes and features to be verified have been successfully tested. Ifthe features have not been tested, it determines what the testgenerators can do to remedy the shortcomings.

Another objective is to provide the capability for dynamically changingtest specifications based on the analysis of tests that are generatedand their contribution to the progress of functional verification (i.e.functional coverage), and to be able to observe, measure and respond toany consequences of any such changes.

Yet another objective is the automatic analysis of the state of a designverification and its progress based on implicit and explicit dataextracted from various design and simulation databases, resulting in theautomatic adjustments and optimizations of the test generation process,simulation environment, as well as the coverage models monitoring andmeasuring the verification progress.

The present invention relates to a system for providing coverageanalysis of functional verification of integrated circuit designs, suchas processor cores, embedded microprocessors, and general purposemicroprocessors. The coverage analysis serves to detect functionalverification trends and patterns and to report the same. The systemcomprises a number of components. These include a test generator thatcreates test programs using machine readable test attributespecifications. The test programs are developed by the test generatorbased on specifications that are received from a test spec generator. Asimulator compares the test programs against the simulated behavior ofdesign models. The inputs to the simulator are based on the developmentof deterministic tests and automatically generated tests. A program isincluded for mining and analyzing the simulation database, the programhaving the capability of providing feedback information for tuning thesedeterministic tests. The simulation database contains the results,traces, and logs of the simulation generated by the verificationprocess. A program is used for mining and, analyzing the simulationdatabase. A database of all test programs generated by the testgenerators and engineers is created. A program is in place for miningand analyzing the test program database. Another database contains allfunctional coverage analysis data and coverage model results. A programis used for mining and analyzing the functional coverage database. Thesystem also utilizes a program for analysis and monitoring of thecoverage models and another program for the automatic generation of testspecs from design HDL. A program is also employed for test specadjustment and optimization. Another program is provided for creatingarchitectural coverage models of the test programs. A functionalcoverage database is employed for storage of the created coveragemodels.

A coverage report is generated based on the information in the coveragedatabase and the result of the analysis and mining of the said databasewherein the actual verification process trends and patterns discoveredare fed back to the test generator, to the coverage analysisenvironment, to the functional coverage database, and to the coveragereport generator. The test spec generator receives feedback from thesimulation traces database mining, the test program database mining, andthe coverage database mining modules.

The system may also include a high level description of the designspecification for the generation of machine readable coverage analysismodels, a finite state machine and design state event extractor, and acoverage model generator using these as input to generate new coverageanalysis models or update existing models.

The invention also includes a method for coverage analysis of functionalverification of integrated circuit designs, such as processor cores,embedded microprocessors, and high performance microprocessors. Themethod includes the step of generating test programs using machinereadable design specifications, and storing the test programs in adatabase. The test programs are created based on test specs. They arealso based on feedback from the analysis and mining of the test programsdatabase, the simulation traces database, and the functional coveragedatabase. In the next step, the test programs are compared against thedesign models to produce simulation traces, and the resultant traces arestored in a database. This is followed by mining and analyzing the testprogram database and the simulation traces database. From this analysis,architectural coverage models and microarchitectural coverage models,are created and are stored in a functional coverage analysisenvironment. Some of the architectural coverage models andmicroarchitectural coverage models are generated manually whereas othermodels are computer generated. The method may further include the stepsof developing deterministic tests, and providing feedback to these,based on the mining and analysis of the simulation traces database. Theanalysis and mining of the coverage database is used to detect andreport trends and patterns of interest for functional verification.Coverage reports are interpreted and made available to user via aplurality of interface devices and medium. Coverage directives aregenerated indicating tasks to be performed requiring certain skills anddesign knowledge. The generated reports, advisories, and directives aredisseminated according to predefined methods and media protocols.

The invention further includes an article of manufacture comprising acomputer usable medium having a computer readable program embodied insaid medium, wherein the computer readable program, when executed on acomputer, causes the computer to perform the following steps:

-   -   a) generate a test spec based upon predetermined design        attributes, and forward the test spec to a test generator; b)        create a test program based on the test spec and store the test        program in a test programs database; c) forward each test        program and manually generated tests to a simulator; d) simulate        the test program and tests against the design model, and        generate a trace of the simulation; e) store the traces in a        simulation traces database; f) mine the simulation traces for        patterns; g) mine data in the test programs database; h) analyze        data received from the test programs database, the simulation        traces database and from the database mining, and generate new        models if required, based on the presence or absence of a model;        and i) store the models in a functional coverage database.

The computer readable program further causes the computer to generatedesign specs, and to generate coverage models based on these specs. Thegenerated coverage models may be used based on the analysis of the testprogram database and the simulation traces database, and the mining ofsaid databases.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will now be described with specific reference tothe drawings in which

FIG. 1 is an overview of the system architecture;

FIG. 2 is the detailed system architecture and functional flow diagram;and

FIG. 3 represents a computer medium useful with the current invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates to a system and a method that uses acombination of database mining algorithms and data analysis andoptimization techniques such as classification and regression trees,neural networks, k-nearest neighbors, regression analysis and clusteringalgorithms to provide readily available and usable verificationintelligence (VI). The proposed system and method turn raw data into VIand provide automatic extraction of the required data and attributesfrom existing verification and simulation databases—thus reducing thedependency on designers, and verification personnel. In addition tostandard database mining algorithms such as classification andregression trees, neural networks, k-nearest neighbors, regressionanalysis and clustering algorithms, Design FSM (Finite State Machine)extraction, design event extraction, coverage model tuning andmanagement and automatic coverage model generation techniques areutilized to extract states, events and functional scenarios from designHardware Description Language (HDL) specification and associated design,simulation and coverage databases. In addition, the proposed systemprovides mechanisms to automate the management and dissemination of VIvia various communication medium (email, pager, Website, reports, . . .). An integrated web-based database access and management portlet allowsa single point of access and analysis to all corresponding databasesacross Intranet and Internet.

Here is a partial list of the features of this system:

-   -   Automatic coverage model generation from design specification        HDL        -   Identify and extract primary design interfaces (Input and            output ports), temporal attributes and functional sequences        -   Identify and extract internal design interface signals,            communication protocol attributes and functional constraints            and dependencies,        -   Classify design signals, attributes and functions based on            design hierarchy references, and . . .    -   Automatic coverage model generation from valid scenarios not        covered by any existing models        -   Dominant patterns and sequences in the coverage database are            converted to models            -   Such models can be used to filter data (valid but not                interesting or relevant)            -   Stop generation of test programs contributing only to                such models to improve test generation and simulation    -   Feedback loop to test generators and simulators to reprioritize        resource allocation    -   Automatic coverage model parameter tuning based on design        changes, actual functional verification progress, discovered        design bugs, and dominant verification trends    -   Fuzzy coverage estimation of incomplete scenarios and scenarios        with partially masked fields (e.g. undecided features,        confidential features not released by customer, . . . )

The proposed system and method converts coverage data into usefulcoverage metrics, coverage models and coverage reports and in turnprovides verification environment management and resource optimization.

Turning now to the drawings, FIG. 1 is a diagram showing the high levelview of a typical system architecture according to the presentinvention. Tests are generated in step 20, and are simulated in step 25.Both the generated tests 20 and the simulation results, traces and logs25 are analyzed in step 30. The results of this analysis, as well as thegenerated tests and simulation results, traces and logs are stored invarious databases 40. These databases include a Test Program Database, aFunctional Coverage Database and a Simulator Traces Database, all shownin FIG. 2. These various databases, tests and coverage models arefurther analyzed and the databases are mined at 50 and tests, coveragemodels and simulation environment parameters are optimized at 60. Theoptimized results are recycled to the test generation and simulationsteps 20 and 25.

FIG. 2 is a diagram showing a typical system architecture according tothe present invention.

The microprocessor functional verification starts with defining andgenerating tests (i.e. test programs). These are programs that are thenexecuted against a reference design model to ensure design'sfunctionality. The attributes and properties of desired tests areprovided as input to a Test Spec Generator (TSG) 110. TSG converts thedesired test attributes to appropriate test specification suitable forthe test generators (random, pseudo random, targeted, . . . ) used inthe verification process. The goal is to generate as much differentvariety of tests as possible to insure that all aspects of a new designhave been verified. The input to TSG may include data that is generatedautomatically as well as data that is manually generated by designers116 or verification engineers 118. Input to TSG can be extractedautomatically from test plans 112 or from the structural analysis of thedesign HDL 114.

The TSG 110 takes all the various inputs from the human designers, thesystems, the documents, etc., and generates a specific Test Spec in aformat suitable for a variety of Test Generators 120 deployed in theverification process.

The Test Generator 120 takes the Test Spec and creates test programshaving certain features or characteristics in order to meet theattributes and properties included in the input Test Spec. Such TestSpecs for example may include data and address ranges, list of desiredattribute types, functions, or constraints. Therefore, a random testgenerator would try to cover the requirements in a series of tests. Forexample, the Test Spec might request collisions for a cache. Testgenerator would create tests that generate read-write, write-read,write-write and read-read collisions for all valid data block sizesacross all valid cache addresses. These are all the different featureswanted in this test. The Test Generator 120 generates a set of testswhich are send to the simulator 128 as well as stored in the TestPrograms Database 124 which basically is a central storage for all ofthe tests generated and simulated.

The test program after it is generated is directed to the Simulator 128on the right hand side of the diagram. Here, the program is simulatedagainst the design model to see whether the design would pass or failthe test. If the design fails the test, then the fail reason must bedetermined.

Another input to the Test Simulator is the Deterministic Test in the box132 above the simulator 128. There are two different types of tests thatgo to the Simulator 128. These are the tests that are automaticallygenerated from the test specs by test generators 120. Also test that aregenerated manually by design or verification engineers to verify aspecific condition, which perhaps can't be generated automatically, orthe engineers don't know how to define the requirements to the testgenerator 120. So in either case the Simulator 128 takes test cases,whether manually generated meaning Deterministic Tests 132 orautomatically generated meaning from the Test Generator 120 andsimulates them.

When a test is simulated, the Simulator 128 generates traces orsimulation logs which include the initial assumptions used forinitiating the simulation session, intermediate steps as eachinstruction in the test is simulated and the final results which wouldinclude all values that changed in design due to the execution of thetest. Corresponding simulation traces and logs for all tests whetherpassed or failed are stored in the Simulation Traces Database 136 forfurther analysis and database mining. Some simulators generate standardformat traces such as Value Change Dump (VCD) and other generatedtextual trace files for each simulation session. The Simulator usuallyinitializes the simulation session to a defined state according to thetest spec or a random initial state and then executes the test and aftereach step (or prescribed actions), the simulator identifies whetheranything has changed in the design state and makes a note of the changesas intermediate design state changes which are usually recorded in thesimulation traces, as well as propagated to the design model. At the endof the simulation session the simulation trace containing all designstate changes, as well any simulation logs containing the fate of thetest are directed to the Simulation Traces Database 136.

Simulation Traces Database Mining module shown in box 140 appliesappropriate database mining and analysis algorithms and methods such ask-nearest neighbors, and clustering algorithms to identify new trends,as well as dominant patterns and attribute sequences in the SimulationTraces Database 136. In addition the most common trace patterns, themost common pattern sequences, and the patterns that happen veryinfrequently are also identified by applying appropriate database miningand analysis algorithms and methods such as k-nearest neighbors,regression analysis and clustering algorithms 140. When looking atmillions of traces and noting that some patterns have only happened afew times, whereas other patterns have happened many times, that meansthat those patterns that have happened very few times are under testedand under utilized. This basically gives insight into what area ofverification and test generation may need more attention. Knowing whattest attributes (simulation traces) have happened infrequently in teststhat have passed but frequently in tests that have failed is a greatinsight for debugging the design, as well as valuable hints on the typeof test attributes that need to be mixed with the generated tests. Andthat is where the Simulation Traces Database Mining box 140 can makemajor contribution to the verification progress. From the patterns thatare observed, conclusions are drawn and feed back to the Test SpecGenerator 110 as Test Spec tuning 142 as well as feed back toDeterministic Tests 132 as Test Suite Tuning 141. Thus, theseverification procedures insure that enough attention has been paid toall aspects of the design because of the tendency to test the easiestaspects thoroughly while giving little attention or no attention at allto the more complex scenarios. And the advantage of applying databasemining and analysis techniques at the simulation trace level is toidentify the actual progress and exhibited behavior of the verificationprocess. Thus, the trace mining report may observe that a given scenariowas tested only a few times even though a large number of simulatedtests should have created that scenario. And if the results are not whatwere intended, then there may be defects with the test generation orTest Spec specification or Test Spec generation. Existing verificationmethodologies would not be able to get such an insight into the actualinner working of the verification process. In addition, the SimulationTraces Database Mining module 140 can identify rare tested conditionsindicating the need for additional class of tests to be generated.

To the left of the Test Programs Database box 124 is a box 144 labeledTest Programs Database Mining. Test specs serve as input to the TestGenerator 120 to generate tests that have required specialcharacteristics based on the test spec inputs. The Test Generator 120does its best to create the tests satisfying all or as many of therequired specifications as possible. Test generators are not always ableto generate the exact required tests because of some conflicts that cannot be resolved or because their internal generation algorithms may notconverge in the permitted time. Test Programs Database mining module144, applies appropriate database mining and analysis algorithms andmethods such as regression analysis and clustering algorithms to theTest Programs Database 124 to determine whether an evenly distributedtest signature is present or whether there are test generation biases(frequent patterns, rare patterns, missing pattern sequences,). Any newdiscovery is fed back to Test Spec Generator 110 in the form of TestSpec Tuning. This will result in updated Test Specs or new Test Specs,which will enable the Test Generator 120 to generate appropriate testpatterns or test features. This feed back and tuning process is directedby the insight into the inner working of the test generator as well asinsight into the design itself. The database mining is usually used foranalyzing and finding relationships between the information in thedatabase, but it can also be used to find a lack of a relationship. TheTest Generator can then be instructed to adjust its generation processby adjusting the test spec fed to it in order to make sure that the typeof tests required are generated. If such a generation falls outside ofthe test generator capability, appropriate advisory reports aregenerated. Thus, even though the test generation might have beensuccessful and the test passed simulation, the Test Programs DatabaseMining 144 can detect if the test does not include all of the requestedfeatures. This test program database analysis is conducted before thetest programs are simulated, which could stop the simulation ofredundant, incorrect, and incomplete tests.

In addition to Test Spec Tuning feed back generated by Simulation TracesDatabase Mining 140 and Test Programs Database Mining 144, coverageanalysis model tuning directives are also generated by both 140 and 144.These Model Tuning directives are fed to Coverage Analysis Environment150. The Coverage Analysis Environment model optimization routines willthen identify whether the received information is applicable toArchitectural Coverage Models 158, Microarchitectural Coverage Models154 or both and make the appropriate adjustments. For example, the ModelTuning directives generated by the simulation traces database mining 140may apply to both architectural and microarchitectural models. ModelTuning directive from Test Programs Database Mining 144 are usuallyapplicable to the Architectural Coverage Models 158.

One of the important features of database mining is pinpointingrelationships that are present in the Test Program Database 124 orSimulation Traces Database 136 but were not planned for nor anticipated.Identifying patterns or attributes in the databases which can not beaccounted for based on the test specs and simulated tests, indicate thatthere are some valid design features missing in the verification plan orperhaps there are some unknown (unanticipated) test generatorcapabilities. Any such discovery will be fed back to Test Spec Generator110, Deterministic Tests 132, as well as Coverage Analysis Environment150.

The Simulation Traces Database Mining 140 and the Test Programs DatabaseMining 144 can be used to make sure not only that planned test andsimulation patterns are present but also to identify patterns that werenot anticipated or planned for such as unusual relationships, sequences,patterns and biases. This information is fed to Coverage AnalysisEnvironment 150 as Model Tuning. Three Model Tuning arrows are shownfrom the Simulation Traces Database Mining 140, the Test ProgramsDatabase Mining 144, and the Coverage Database Analysis Mining 166. Allof these are feedbacks from the database mining and database analysis.If a scenario, feature or relationship is found that was not initiallymodeled, meaning that it may have been overlooked, it is submitted tothe modeling environment 150, so a model specification is generatedwhich would result in either automatic generation of one such coverageanalysis model or a request is sent to the responsible verificationengineer. This may result in upgrade of an existing model to ensureadditional scenarios, feature or relationship is monitored by the modelor new coverage analysis models may be required. In some instances, sucha feedback would identify non-functional models—models that should havecaught the listed scenario, design feature or feature relationship butdid not because of incorrect modeling or wrong assumptions. In someinstances, the scenario, design feature or feature relationship arecoverage exceptions (Features that are not implemented in the design andtherefore should be excluded in the coverage analysis and should havenever been simulated successfully). This class of discovered attributesare in fact design errors that escaped the verification processindicating an error in the simulation model. Thus, the database miningin the present invention is used to find scenarios, relationships andfeatures that are known, or that should exist. It is also used toidentify if these in fact have occurred and are present in one of thedatabases. Database mining is also used to identify relationships,scenarios and patterns that have not been contemplated or that shouldnot happen. All three cases again are sent back into the modelingenvironment 150 with some of the model tuning directives only applyingto architectural models 158 and some of them applying only tomicroarchitectural models 154, while others may apply to both.

For purposes of the present invention, it should be understood that theArchitectural Coverage Models 158 deal with the architectural featuresof the design, such as instruction set, register set, caches, pipelines,data ranges, address ranges, data conversion features, branch managementfeatures, etc. The Microarchitectural Coverage Models 154 on the otherhand address the underlying micro operations and functions.Microarchitectural coverage models 154 ensure these internalsubcomponents and hidden layers of the design are verified. These modelsmonitor the inner working and temporal dependencies of the design.

The Architectural Coverage Models 158 mainly apply to the tests(automatically generated as well as deterministic tests) whereas themicroarchitectural coverage models 154 mainly are applied to simulationresults of the tests which includes more detailed internal working ofthe design, signal values and system status condition changes.

Coverage analysis environment 150 applies coverage models to testscoming to Test Program Database 124, as well as simulation traces anddumps coming to Simulation Traces Database 136. The results generated bythe models as well as analysis logs and traces generated by the CoverageAnalysis Environment 150 are stored in the Functional Coverage Database162. The information stored in 162 includes list of models fired(activated) for each test, components of each model fired for each test,and incorrect features observed by a model (exceptions that should havenot happened). In addition, each model has a corresponding set ofcounters keeping track of the number of times the model and its specificcomponents have been activated. For example, if model A has 10components (10 design features or scenarios to be tracked), there willbe at least 11 counters associated with it (model-a, a1, a2, a3, . . .a10), where model_a and a# counters are incremented each time a modelattribute (a#) is observed. In addition there will be an associatedcross reference table which would include the name of the tests thattriggered the model. Some implementations may only save the name of thefirst and last tests, to manage the size of the corresponding databases.All such relationships and information (depending on the deployedmethodology) will be maintained in the Functional Coverage Database 162.

In addition to the coverage analysis models 150 which are applied toTest Programs Database 124 and Simulation Traces Database 136, there arealso a set of models and filters used for analysis of FunctionalCoverage Database 162 content. The Coverage Database Analysis and MiningModels, Rules, Filters and Attributes 166 analyze and evaluate theFunctional Coverage Database 162 itself. In addition to applyingappropriate database mining and analysis algorithms and methods to thisdatabase similar to functions performed by 140, 144 and 164, CoverageDatabase Analysis and Mining 166 also performs maintenance andoptimization routines on this database. For example, it can identifymodels that have very similar signature which may indicate duplicate orredundant models. It can also identify models that complement each other(i.e. logical intersection of model A and model B is null), which canidentify models that can be combined. The Coverage Database Analysis andMining module 166 can also identify frequently hit models which mayindicate models that are very shallow or too general. The function 166also analyzes database 162 to identify correlations, dependencies andbehavioral trends and associations among models. The result of CoverageDatabase Analysis 166, as well as the result of Coverage Model DatabaseMining 164 are fed back to the Coverage analysis Environment 150 forcoverage model adjustment and optimization, as well as removingredundant models and creating missing ones. In addition, the result ofCoverage Database Analysis 166 is also fed back to the Test SpecGenerator 110 for adjustments to Test Specs and to Coverage ReportGenerator 170 to generate appropriate advisories and directives foractions that require attention from engineers. The model Tuningdirectives generated by 166 will guide 150 to make necessary operationaladjustments in response to the behavior and patterns observed in 162.For example, if one hundred models are maintained in the Database 162and 95 of these 100 models have been fired (activated) very frequently,but 5 of them have hardly been activated, this indicates that there areinherent biases in the test generation process that have ignored thedesign attributes monitored by these neglected models or there might besome problems with the Test Spec Generator 110 or perhaps missing orinadequate Test Specs. Therefore, resources and expertise should bedevoted to this set of models and their corresponding tests andverification tasks. A closer investigation is required to identify thecause of this neglect. Maybe there are not enough tests for it, maybe itis in a part of the design that requires a special feature to beexercised, or requires specific sequences of functions and system statusconditions. Sometimes such a discovery points to weaknesses in theverification methodology where some aspects of the design are not paidenough attention. In other times, it may indicate design features thatwere added but not documented, or specific implementation detail notavailable in the design specification.

Therefore, as it has been described in the previous sections, databasemining techniques are used to identify aberrations in the testgeneration, simulation, and coverage analysis tasks in order to bring tofocus parts of verification process that need attention. That is one ofthe strength of database mining which is missing from the traditionalfunctional verification and coverage analysis tools and methodologies.And again this is important because these are high level processbehavior that are not reflected in the coverage models or simulationresults. Coverage models usually identify and count items of interestbut can not provide any insight into what might have happened in thescope of the entire verification process. Simulation can only tell uswhether a test failed or passed and does not keep track of previoussimulations or incomplete tasks. Using database mining techniques inconjunction with the traditional model-based functional coverageanalysis provides a deeper insight into the overall verificationprogress, as well as the not so obvious biases and shortcomings of theverification methodology.

The coverage models by themselves do not keep track of all relevant andperipheral events that might have happened simultaneously in conjunctionwith the events they are tracking. Nor do they monitor or can monitorwhether anything unexpected or illegal also happened in other models.And that is where the database mining techniques can make uniquecontribution to improving the overall verification process efficiencyand quality.

The Coverage Report Generator 170 extracts information from theFunctional Coverage Database 162 and generates statistical reports,usage reports, frequency reports and many other coverage reports. Italso takes input from the Coverage Database Analysis . . . 166 togenerate correlation, dependency, and relationship reports for thecoverage models. All of these coverage reports and others that can begenerated from 166 and 162 are then made available to engineers via theWeb-based Graphical User Interface Display Portlet 174. This Portletsimilar to a Java applet in concept provides a programmable facility fordynamically generating and customizing display objects, such as graphs,images, charts, etc. Portlets are Java-based Web components, managed bya portlet container, that process requests and generate dynamic content.The disclosed system and method uses portlets as programmable userinterface components that provide a presentation layer to the compiledand the generated information. This Display Portlet 174 can be accessedfrom internet as well as intranet and will automatically customize thedata and display objects for the user preferences and requirements, suchas language, display schema, colors, presentation type (audio, video,text, . . . ), etc. providing a modular and user-centric datapresentation mechanism. 174 also provides support for remote and local,as well as wired or wireless user devices. It also provides interactionprotocols for future interface device integration and support. Inaddition, 174 provides mechanisms for user security and identitymanagement, as well as On-Demand information generation anddissemination.

Coverage reports and analysis data produced by generator 170 areanalyzed by the Coverage Report Interpretation 178 in the context of theverification progress, available user profiles, types of verificationdirectives available, project profile and the available expertise(engineers). The Coverage Report Interpretation 178 generates specificdesign and verification directives on who should do what task, usingwhich tools and how. The generated design and verification directivesare then disseminated to appropriate engineers based on the predefinedrules and protocols 182.

An yet another aspect of this disclosure is the methods by which some ofthe coverage models and test specifications can be generatedautomatically. The Design Spec HDL 186 is the design specification in asuitable HDL language or format such as VHDL, Verilog, System Verilog,SystemC, etc. From structural analysis of the Design HDL 186 somearchitectural and microarchitectural features of the design can beextracted and converted to Test Specification 114. In other words, oncea design feature is extracted from Design HDL 186, it can be used as aTest Spec hint or requirement input 114 to Test Spec Generator 110. Inaddition, methods are deployed that would extract (or estimate) a designFSM (Finite State Machine), state tables, and functional events 190automatically from the Design HDL 186. Then the corresponding orderedlist of events, the sequence of events, and the list of forbidden eventscan be generated and made available to the Coverage Model Generator 194.These are coverage analysis models that are generated automatically andare then passed to either the Architectural Coverage Models 158 orMicroarchitectural Coverage Models 154 modules. This is in addition toall the models generated manually by design and verification engineers.Some of the coverage models can only be defined by the design andverification engineers who have intimate knowledge of the design intentand its specific implementation. And some of the coverage analysismodels can be generated automatically as described above. Thiscapability is specially beneficial when the design specs change, and thedesign or verification engineer may not be up to date with the changesin the design specs because of the changes initiated by the customer,marketing or the architect. So, such an automation provides mechanismsto insure that if there are new features added to the design spec, orany design spec changes, that they are taken into consideration andcorresponding models are generated to encompass the changes and thesemodels are placed into the coverage model analysis environment 150.

Tests are generated in step 20, and are simulated in step 25. Both thegenerated tests 20 and the simulation results, traces and logs 25 areanalyzed in step 30. The results of this analysis, as well as thegenerated tests and simulation results, traces and logs are stored invarious databases 40. These databases include a Test Program Database, aFunctional Coverage Database and a Simulator Traces Database, all shownin FIG. 2. These various databases, tests and models are furtheranalyzed and the databases are mined at 50 and tests, coverage modelsand simulation environment parameters are optimized at 60. The optimizedresults are recycled to the test generation and simulation stepsgenerator steps 20 and 25.

While the invention has been described in combination with specificembodiments thereof, there are many alternatives, modifications, andvariations that are likewise deemed to be within the scope thereof.Accordingly, the invention is intended to embrace all such alternatives,modifications and variations as fall within the spirit and scope of theappended claims.

1. A system for providing coverage analysis of fractional verificationof integrated circuit designs comprising: a) a test specificationgenerator that receives feedback from simulation traces database mining,test program database mining, and coverage database wining modules andgenerates machine readable test attribute specifications based on saidfeedback; b) a test generator that creates test programs using machinereadable test attribute specifications, wherein the test programs aredeveloped by the test generator based on specifications that arereceived from said test specification generator; c) a simulator thatcompares the test programs against design model simulations; d) adatabase containing traces of simulations generated by a functionalverification; e) a program for mining and analyzing the simulationtraces database; f) a database of all test programs generated by thetest generator, and by designers and verification engineers; g) aprogram for mining and analyzing the test programs database; h) adatabase of analysis logs and traces and coverage model results; i) aprogram for mining and analyzing the functional coverage database; j) aprogram for analysis and monitoring of coverage models; k) a program forautomatic test spec generation from design HDL; l) a program for testspec adjustment and optimization; m) a program for creatingarchitectural and microarchitectural coverage models of the testprograms; and n) functional coverage database for storage of the createdarchitectural and microarchitectural coverage models.
 2. A system forproviding coverage analysis of functional verification of integratedcircuit designs comprising: a) a test generator that creates testprograms using machine readable test attribute specifications; b) asimulator that compares the test programs against design modelsimulations and stores the simulations in a database, wherein the inputto the simulator is based on the development of deterministic tests; c)a database containing the traces of simulations generated by averification process; d) a program for mining and analyzing thesimulation database; e) a program for mining and analyzing thesimulation database including the capability of providing feedbackinformation for timing the deterministic tests; f) a database of alltest programs generated by the test generator, and by designers andverification engineers; g) a program for mining and analyzing the testprograms database; h) a database of all functional coverage analysisdata and coverage model results; i) a program for mining and analyzingthe functional coverage database; j) a program for analysis andmonitoring of the coverage models; k) a program for automatic test specgeneration from design HDL; l) a program for test spec adjustment andoptimization; m) a program for creating architectural andmicroarchitectural coverage models of the test programs; and n)functional coverage database for storage of the created coverage models.3. A system for providing coverage analysis of functional verificationof integrated circuit designs for the purpose of detecting verificationtrends and patterns to report the same, comprising: a) a test generatorthat creates test programs using machine readable test attributespecifications, b) a simulator that compares the test programs againstdesign model simulations; c) a simulation database containing traces ofthe simulations generated during detection of verification; d) a programfor mining and analyzing the simulation database; e) a test programsdatabase of all test programs generated by the test generator, and bydesigners and verification engineers; f) a program for mining andanalyzing the test programs database; g) a database of all functionalcoverage analysis data and coverage model results; h) a program formining and analyzing the functional coverage database; i) a program foranalysis and monitoring of coverage models; j) a program for automatictest spec generation from design HDL; k) a program for test specadjustment and optimization; l) a coverage analysis program for creatingarchitectural and microarchitectural coverage models of the testprograms; m) a functional coverage database for storage of the createdcoverage model, and n) a coverage report generator based on theinformation in the functional coverage database and the result of theanalysis and mining of the said database wherein the actual verificationprocess trends and patterns discovered are fed back to the testgenerator, to the architectural and microarchitectural models, to thefunctional coverage database, and to the coverage report generator.
 4. Asystem for providing coverage analysis of functional verification ofintegrated circuit designs comprising: a) a test generator that createstest programs using machine readable test attribute specifications; b) asimulator that compares the test programs against design modelsimulations; c) a database containing traces of simulations generated bya verification process; d) a program for mining and analyzing thesimulation database; e) a database of all test programs generated by thetest generator and by designers and verification engineers; f) a programfor mining and analyzing the test program database; g) a database of allfunctional coverage analysis data and coverage model results; h) aprogram for mining and analyzing the functional coverage database; i) aprogram for analysis and monitoring of coverage models; j) a program forautomatic test spec generation from design HDL; k) a program for testspec adjustment and optimization; l) a coverage analysis program forcreating architectural and microarchitectural coverage models of thetest programs; m) functional coverage database for storage of thecreated coverage models; n). a high level description of the designspecification for the generation of machine readable coverage analysismodels; and o) a finite state machine and design state event extractor,and a coverage model generator using said machine readable coverageanalysis models as input to generate new coverage analysis models orupdate existing models.
 5. A method for coverage analysis of functionalverification of integrated circuit designs comprising the steps of: a)generating test programs based on 1) optimized test specifications andon 2) feedback from analysis and mining of a test programs database, asimulation traces database, and a functional coverage database usingmachine readable design specifications, and storing the test programs ina database; b) comparing the test programs against design modelsimulations to produce simulation traces, and storing the resultanttraces in a database; c) mining and analyzing the test programs databaseand the simulation traces database; d) creating architectural coveragemodels microarchitectural coverage models; and e) storing the createdmodels in a functional coverage analysis database; and f) developingdeterministic tests, and providing feedback to these deterministictests, based on the mining and analysis of the simulation tracesdatabase. The above changes correct informalities and accurately definethe claimed invention.